Add database setup #74

fhenneke · 2024-10-08T10:42:52Z

This PR adds a setup file for the database and a docker image to create a local database.

The database is set up to have four tables

A table with settlement transaction hashes and timestamps.
A table with settlement transaction hashes and token addresses. It should be populated with all tokens traded in a settlement. Together with the first table we have all tokens traded with times when they were traded.
A table for prices from different sources.

This table structure will be mirrored once per chain.

The database can be set up locally using the commands in the README or using the make file command make test_db.

The code does not use this table layout yet. I am planning to add another PR which uses those tables instead of the current tables raw_token_imbalances, slippage_prices, and fees_new.

harisang · 2024-10-08T11:07:07Z

database/01_table_creation.sql

@@ -0,0 +1,23 @@
+CREATE TABLE token_info (


Shouldn't we make it chain-specific? I would rename it to

token_info_mainnet

given that we will probably use one version of the db for all chains

Why would we have one database for all chains instead of one database per chain? We use the latter for all other purposes.

If we go with having information on all chain in one database, I would expect that having an additional column network in all tables is better that duplicating tables for each chain.

Why would we have one database for all chains instead of one database per chain? We use the latter for all other purposes.

Don't have a strong opinion here, but this would add overhead, as we (i.e., devops) would need to set up different databases, accounts etc

If we go with having information on all chain in one database, I would expect that having an additional column network in all tables is better that duplicating tables for each chain.

Although postgres most likely takes care of everything, having a single table where multiple daemons might try to add things at the same time could create some race conditions. Also, could it make things slower, as network should be part of the primary key now?

Somehow having a separate table per chain looks safer/more flexible.

What do you think, @ahhda?

harisang · 2024-10-08T11:08:04Z

database/01_table_creation.sql

@@ -0,0 +1,23 @@
+CREATE TABLE token_info (
+    token_address bytea PRIMARY KEY,
+    symbol varchar NOT NULL,


Tbh the symbol is not needed, so i would remove it (as i am also always nervous with the crazy characters some tokens use that might cause encoding issues here). And if you want to include it, i would definitely make it optional

Then lets just remove it. No need to store unused data here.

harisang · 2024-10-08T11:19:29Z

database/01_table_creation.sql

+    decimals int NOT NULL
+);
+
+CREATE TABLE token_times (


I find the name a bit unintuitive. I would probably change it to something like

transaction_timestamps

or imbalance_timestamps

Would token_timestamps work? Or token_transactions?

This table is not about imbalances and ~~it is not about transactions~~ (edit:) it is not about timestamps of transactions but about tokens. So not having token in the name sounds misleading to me. (end edit) The table might be superseded by a token_imbalances table for linking tokens to transactions, maybe in combination with a settlements table linking transaction hashes and times.

This table is not about imbalances and it is not about transactions.

Hm, then what is it about? There is a tx hash associated with each entry

I added a clarification in the comment above.

For a table with the name transaction_timestamps i would expect that (tx_hash, time) is a key. Having token addresses in that would surprise me.

For a table with the name imbalance_timestamps i would expect that it stores imbalances in some form.

Maybe the clean design would be to have a transaction_timestamps table and a transaction_tokens table. To avoid overhead due to additional tables it could also be a transaction_token_timestamps table with columns (tx_hash, token_address, time) or something like that.

Yeah, the distinction between the transaction_timestamps and transaction_tokens tables make sense. And actually we could easily generate the transaction_tokens table already by reducing the scope of the part of the code that computes imbalances (it could simply just keep track of what tokens are being transferred and record those in this table). I.e., repurpose the raw_token_imbalances table into a transaction_tokens table until further notice.

bram-vdberg

LGTM! I added a makefile so we can start the database easier.

Would it be worth adding an index? It would make querying specific tokens faster:

CREATE INDEX idx_prices_token_time ON prices (token_address, time);

harisang

Definitions look good! I cannot really comment on the rest

harisang · 2024-10-09T15:58:47Z

database/01_table_creation.sql

+    PRIMARY KEY (tx_hash, token_address)
+);
+
+CREATE TYPE PriceSource AS ENUM ('coingecko', 'moralis', 'dune', 'native');


@fhenneke Do you know if this is easy to change, in case we end up using some other price feeds as well?

fhenneke added 2 commits October 8, 2024 12:36

add file for database creation and docker file for local database

d188824

add some comment on local setup of database

fbced8a

fhenneke requested review from bram-vdberg and harisang October 8, 2024 10:42

harisang reviewed Oct 8, 2024

View reviewed changes

Add Makefile for easy db building

a14969d

bram-vdberg approved these changes Oct 8, 2024

View reviewed changes

restructure tables

4fe802b

harisang approved these changes Oct 8, 2024

View reviewed changes

fix naming

8f55892

fhenneke marked this pull request as ready for review October 9, 2024 13:12

Merge branch 'main' into add_new_database

567fd0b

harisang reviewed Oct 9, 2024

View reviewed changes

harisang merged commit 21727e4 into main Oct 10, 2024
3 checks passed

fhenneke deleted the add_new_database branch October 10, 2024 09:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add database setup #74

Add database setup #74

fhenneke commented Oct 8, 2024 •

edited

Loading

harisang Oct 8, 2024

fhenneke Oct 8, 2024

harisang Oct 8, 2024

fhenneke Oct 8, 2024

harisang Oct 8, 2024 •

edited

Loading

fhenneke Oct 8, 2024

harisang Oct 8, 2024 •

edited

Loading

fhenneke Oct 8, 2024 •

edited

Loading

harisang Oct 8, 2024 •

edited

Loading

fhenneke Oct 8, 2024

harisang Oct 8, 2024

bram-vdberg left a comment

harisang left a comment

harisang Oct 9, 2024

Add database setup #74

Add database setup #74

Conversation

fhenneke commented Oct 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

harisang Oct 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

harisang Oct 8, 2024 • edited Loading

Choose a reason for hiding this comment

fhenneke Oct 8, 2024 • edited Loading

Choose a reason for hiding this comment

harisang Oct 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bram-vdberg left a comment

Choose a reason for hiding this comment

harisang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fhenneke commented Oct 8, 2024 •

edited

Loading

harisang Oct 8, 2024 •

edited

Loading

harisang Oct 8, 2024 •

edited

Loading

fhenneke Oct 8, 2024 •

edited

Loading

harisang Oct 8, 2024 •

edited

Loading